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• Real-Time Cutscenes 

• Hand Keyed Animations 

• Joint Only Face Rigs 
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• Impactful BackStory 

* Realistic Characters and Animations 

• Facial Motion Capture 

* Bodies/Faces/Eyes/Audio -Simultaneous 

* Multiple Actors -Simultaneous 

• Real-time Cutscenes 
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• Hand Keyframed 

• 3d Scan Capture Per Frame Performance 

• Muscle-Based/Simulation 

• Data Driven/Example Based 
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• Requires a Data Set of known example input to ouput 

• Enough data to make a generalized representation 

• Data-Set 

• Facial Scans (F.A.C.S) (Paul Ekman) 

• Consistent Facial Marker Set (scans/mocap) 

• New Inputs 

• Tracking (same marker set) 

• Solver (Applies Mocap Data to Rig/Shapes) 

• Motion Retarget to Game Characters 








anim tweaks 
sticky lips deformer 


anim export 
+ vertex system 




live game engine render 


body + facial animation rig 
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live in-engine wrinkle 











motion capture (HMR) 


body + facial animation rig 


live in-engine wrinkle 
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anim tweaks 
sticky lips deformer 


live game engine render 


pose space deformer 
regions + per joint 


retarget to game char 


pose space training 











• Pre-Captured (One time setup cost) 

• 70 Facial Pose Scans per actor 

• Scan Re-mesh to consistent Topology 

• Scan Hardware Used 

* 2 linked structured light field Scanners 

• Captured in Static Volume 

• Stabilize/Align Scans 
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* Consistent Topology/Blendable 

* Consistent Marker Correspondence 

* Custom Vacuform Masks 

* Track Markers/Warp Fit/ Subdivide/Snap 

* Combine Faces with Entire Head 

* Model Tweaks (Noisy/Missing Scan Data Eyes/Mouth) 
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• Head Mounted Camera Rigs (HMR) 

• Simultaneous Capture - Face/Body/Eyes/ Audio 

• Timecode Hardware Sync 

• Multiple Actors 

• Projection Mask Animation 

• Marker Tracking 
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• Methods (2d vs 3d) 

• Optical Flow 

• Feature Recognition 

• Triangulation Software 

• Noise Reduction 

• Stabilization 

• Moving helmets/Day to Day/Shot to Shot align 

• Missing Marker Data/ Occlusion 

• Lighting Conditions 

S3 










• Methods 

• Game Character Face Rig, trained to use Drivers from actor for 
Blend Weights 

• Transfer Blend Weight results (Shape Weight Correspondence 
between character poses if needed) 

• Hint Based Scalars On Base Scaffold 

• Proportion Constraint (method we used) 

• Shape Re-target 

• Marker Motion Re-target 

• Easily Tie motion to joints 
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motion ca pture (HMR) 


body + facial animation rig 
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3D marker tracking 


retarget to game char 



pose space deformer 
regions + per joint 



pose space training 


anim tweaks 
sticky lips deformer 
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retarget to game char 









• Linear BlendShapes 

• Easy to Animate Weights 

• Over-Adding Issues-need to normalize 

• “Posey” results 
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DATA-DRIVEN SOLVER 
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• Pose Space Deformation (PSD) as solver 

* A method of deriving shape weights from marker data 

* Uses Radial Basis Function (RBF) 

* Applied as Deformer node, shapes driven indirectly though 
a face rig- controls will get automatic shape interpolation 

* Markers as Drivers / Motion Vectors 

* Setting up Matrix/ Single Value Decomposition (SVD) 

* Shapes Converted to PreSkinCluster - Relative to Default 
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Actor 

BlendShape 





Game Char 
SkinCluster 
&PSD 


Game Char 
BlendShape 
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DATA-DRIVEN PSD SOLVER 
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• Fine Tuning PSD Attributes 

• Input Width Regularization 

• Balance Pose Space (Gaps/Shape Overlap) 

• Drop poses too close 

• Testing Drivers - ROM test/ BlendShape test 

• Expose Solver Attributes to Animators 

• Using Multiple PSDs with normalized deformer weights 

• Fewer Drivers for Independent Regions 

• Like Weighted PSD 



SUCKER PUNCH 



S3 



i. » i 


i U * 


DATA-DRIVEN PSD SOLVER 
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• Pose Training Phase (One Time) - surface locators 

• PSD Deformer applied Pre-SkinCluster 

* Joints per Facial Marker 

• Track Drives Skin and Vertex Level Deformations 

• Efficient Streaming - No Dense Shape Memory 

* Smooth Natural Arcs in Motion 



TRAINED POSES 


MARKER PATH 




POSE SPACE PATH 





• 2D versus 3d Tracking 

* 2d requires only 1 camera per head rig 

* 3d requires 4 cameras per head rig 

* 3d track can use the 3d data to verify accuracy 

* 3d track gives better visualization/sillhouette 

* 3d track can be used to warp/fit results tighter 

* 3d track provides extra dimensionality for lips 
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stabilize 





subdivide + snap 
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3D marker tracking 


retarget to game char 


pose space deformer 
regions + per joint 




anim tweaks 
sticky lips deformer 
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anim export I 
+ vertex system | 











• Variety of Animation Controls (Gross and Fine Control) 

• Gross Controls -Directly Manipulate Key Drivers 

• Fine Controls -Directly Manipulate each joint 

• Ul - Picture per actor pose with multiple Region Sliders 

• Automatic Shape Interpolation with all controls 
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• Animator Attributes 

* RBF, input widths, Shape Influence, Drop Shapes 

* Per Joint control weight -From Pure Pose Space to exact 3d 
track 

* Additive versus Animation Over-takes Mocap automatically 

* Sticky Lips Attributes 

* Eye Drag Multiplier Attributes 



SUCKER PUNCH 



motion capture (HMR) 
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body + facial animation rig 



3D marker tracking 
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pose space deformer 
regions + per joint 



pose space training 



retarget to game char 









• Compression and Stretch - Independent across face 

• Detects Local Volume changes relative to Default 

• Low Res Cage associates connectivity of high res 

* Faster Calculation 

* Smoother Falloff 

* Art Direct which Areas to be Driven Together (cage shape) 

• Created the Normal Maps using depth difference 
between Raw dense FACS scans and final resolution 






subdivide + snap 




model edits [lips + eyes] 




motion capture (HMR) 


body + facial animation rig live in-engine wrinkle 



pose space deformer 
regions + per joint 



pose space training 


retarget to game char 









• Deformer Applied Pre-SkinCluster - Better Compression 
with small range offsets 

• Joint Animation as Animation LO.D. 

• GPU Decompression during streaming 

• Normal Recalculation Required 

• Seam Fixing (shared edges) 

* Snag GPU round-off versus Maya skin Weights 

• In Engine Wrinkle/Crease - matches anim rig 
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• Tracking Issues 

* Stabilization, Camera Calibration 

* Requires Fast Feedback loop applied to “actor-only Rig”, 
for iteration to verify track 

• Retarget - some room for improvement 

• Infer Jaw Motion/Teeth - aimed at center of two lower 
jaw markers-(not perfect so allow offset animation) 

• Attention to deformations of neck where join PSDs 
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Josh Jersild - Senior Programmer 

Shuai Liu - Senior Programmer 
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THANK YOU!! 

...And special Thanks to the GDC Committee! 
...and GDC mentor Ru Weer 


Questions? 

email: spencera@suckerpunch.com 
twitter: spensa3d 
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